Fault Tolerant Computing on the Grid: What are My Options?

High-performance distributed computing across wide-area networks has become an active topic of research [1][3][4][11]. Metasystem and grid software infrastructure projects, most notably, Legion [4] and Globus [3], have emerged to support this new computational paradigm. Achieving large-scale distributed computing in a seamless manner introduces a number of difficult problems. This paper examines one of the most critical problems, fault tolerance. A large wide-area system that contains hundreds to thousands of machines and multiple networks has a small mean time to failure. The most common failure modes include machine faults in which hosts go down and get rebooted, and network faults where links go down. A single monolithic solution for fault tolerance that is acceptable to all user applications is unlikely. For example, some applications may require continuous availability, or may require protection from byzantine failures, or require light-weight, low overhead fault tolerance. The most appropriate method for fault tolerance clearly may be application-specific. This follows the current trend in distributed systems and operating systems in which generic functions once performed within the “system” are now being moved to user-space for increased flexibility and performance. Because general purpose systems often impose a high cost on applications that do not fit their assumptions, the maxim “pay for what you need” has been proposed as a guiding principle for application-centric policy decisions in metacomputing systems such as Legion. We believe that the relative performance of fault tolerance methods is a key piece of information needed to enable users to make these decisions on behalf of their applications. This is particularly true for high-performance applications. To this end, we have examined fault tolerance options for a common class of high-performance parallel applications, single-program-multiple-data (SPMD). Performance models for two fault tolerance methods, checkpoint-recovery and wide-area replication, have been developed. These models enable quantitative comparisons of the two methods as applied to SPMD applications. While these